1 link tagged with all of: alignment + multimodal + representation learning + platonic representation hypothesis + jepa
Click any tag below to further narrow down your results
Links
This arXiv paper argues that models trained on different data types can converge on a shared representation, and that it can be proven under specific conditions, not just hoped for as scale grows. Its central finding is that the "good views" used to train joint-embedding predictive architectures, not model size alone, are what drive that alignment.
- The authors prove that JEPAs trained separately on different modalities learn the same representation up to a rotation, provided their positive pairs are two views of the same scene.
- Next-token prediction is framed as a JEPA with "bad views" (parts of one document), which they argue explains why language models align with vision only through scale.
- A 109M-parameter text JEPA aligns with DINOv2 and CLIP image encoders better than every language model tested up to 72B parameters, and needs 42 times fewer samples than next-token prediction to reach the same alignment.